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Integrated  Modeling  of  the 
Battlespace  Environment 

The  goal  of  the  Battlespace  Environments  Institute  (BEI)  is  to  integrate  Earth  and  space 
modeling  capabilities  into  a  seamless,  whole- Earth  common  modeling  infrastructure  that 
facilitates  interservice  development  of  multiple,  mission-specific  environmental  simulations 
and  supports  battlefield  decisions,  improves  interoperability,  and  reduces  operating  costs. 


Characterizing  the  natural  environ¬ 
ment  is  crucial  to  US  Department 
of  Defense  (DoD)  mission  planning 
because  understanding  battlespace 
conditions  enhances  both  safety  and  war  fighting 
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effectiveness.  Historically,  DoD  production  cen¬ 
ters  have  used  stand-alone  models — such  as  those 
for  weather  and  ocean  conditions — which  have  as¬ 
sociated  maintenance  costs.  Although  such  mod¬ 
els  continue  to  improve,  alone  they  can  provide 
only  an  incomplete  representation  of  the  envi¬ 
ronmental  conditions  that  might  impact  a  DoD 
mission. 

Environmental  processes  interact  on  multiple 
time  scales,  and  many  such  processes  interact  on 
time  scales  that  are  short  enough  to  be  signifi¬ 
cant  to  the  DoD.  Our  environmental  subsystems 
therefore  must  be  coupled  into  a  larger  interact¬ 
ing  system.  I'he  problem,  however,  is  that  any 
single  service  lacks  adequate  resources  to  develop 
a  complete,  coupled  prediction  capability  for  the 
battlespace  environment. 

Creating  coupled  modeling  systems  using  a 
standard  DoD  modeling  framework  will  foster 
collaborative  efforts  throughout  the  DoD  and  fa¬ 
cilitate  partnerships  with  outside  organizations. 
VVe  established  the  Battlespace  Environments 
Institute  (BEI)  with  a  vision  of  inulti-agcncy  and 
multiservice  collaboration  for  the  rapid  develop¬ 
ment  and  transition  of  new  models  to  support 
mission  planning,  BEI  stakeholders  include  the 
US  Navy,  Air  Force,  and  Army;  National  Aero¬ 
nautics  and  Space  Administration  (NASA);  De¬ 
partment  of  Flnergy;  Department  of  Commerce; 
and  National  Science  Foundation. 

BEFs  goal  is  to  integrate  Earth  and  space  model¬ 
ing  capabilities  into  a  seamless,  whole-Rarth  com¬ 
mon  modeling  infrastructure  to  allow  interservice 


36 


THIS  ARTICLE  HAS  BEEN  PEER-REVIEWED. 


Computing  in  Science  h  Engineering 


development  of  multiple,  mission-specific  en¬ 
vironmental  simulations  to  support  battlefield 
decisions,  improve  interoperability,  and  reduce 
operating  costs.  To  develop  a  whole-Earth  infra¬ 
structure  that  excluded  components  originating 
outside  the  DoD  would  be  cost  prohibitive.  Given 
that  the  environmental  community  had  already 
invested  heavily  in  the  Earth  System  Modeling 
Framework  (www.earthsystemmodeling.org),  the 
DoD  decided  to  not  only  use  ESMF  as  the  basis 
for  a  common  modeling  infrastructure,  but  also 
to  invest  in  ESMF  to  address  DoD-specific  needs. 

ESMF  provides  the  basic  software  layer  for 
implementing  a  whole-Earth  environment.  How¬ 
ever,  it  doesn’t  mandate  how  components  inter¬ 
act.  A  major  BEI  task  has  thus  been  to  design  and 
implement  rules  that  a  component  must  follow 
to  be  part  of  the  DoD  whole-Earth  system.  To 
encourage  progress  in  this  area,  BEI  has  focused 
resources  on  a  few  projects  aimed  at  developing 
prototypes  of  restricted  domains  (such  as  littoral 
and  air-ocean  environments).  It  will  then  expand 
the  projects’  experiences  to  a  whole-Earth  envi¬ 
ronment  modeling  capability. 

Here,  we  offer  an  overview  of  the  ESMF  soft¬ 
ware  architecture  and  design  strategies,  and  then 
highlight  specific  BEI  projects  focused  on  various 
whole-Earth  system  subdomains. 

Earth  System  Modeling  Framework 

I'hc  ESMF  is  open-source  software  for  building 
climate-  and  weather-related  modeling  compo¬ 
nents  and  coupling  them  together  to  form  ap¬ 
plications.  ESMF  was  motivated  by  the  desire  to 
exchange  modeling  components  among  centers 
and  to  reduce  costs  and  efforts  by  sharing  codes. 
Existing  software-framework  efforts  heavily  in¬ 
fluenced  ESMF’s  design.*  The  project  is  distin¬ 
guished  by  its  strong  emphasis  on  community 
governance  and  distributed  development,  and  by 
a  diverse  customer  base  that  includes  modeling 
groups  from  universities,  major  US  research  cen¬ 
ters,  the  US  National  Weather  Service,  the  DoD, 
and  NASA.  The  ESMF'  development  team  is  cen¬ 
tered  at  the  US  National  Oceanic  and  Atmospheric 
Administration  (NOAA)  Cooperative  Institute 
for  Research  in  Environmental  Sciences. 

Architectural  Overview 

I'he  ESMF  architecture  is  based  on  the  com¬ 
ponents  concept.  At  its  simplest,  a  software 
component  is  a  code  with  a  well-defined  calling 
interface  and  a  coherent  function.^  Component- 
based  design  is  a  natural  fit  for  Earth  system 
modeling  because  components  are  ideally  suited 


to  represent  a  system  with  substantial,  distinct, 
and  interacting  domains — such  as  atmosphere, 
land,  sea  ice,  and  ocean.  Further,  because  Earth 
system  domains  are  often  studied  and  modeled  as 
collections  of  subprocesses  (radiation  and  chemis¬ 
try  in  an  atmosphere,  for  example),  it’s  convenient 
to  model  Earth  system  applications  as  a  nested- 
components  hierarchy. 

Component-based  software  is  also  well  suited 
to  the  way  Earth  system  models  are  developed 
and  used.  Individual  specialists  typically  develop 
a  model’s  multiple  domains  and  processes  as  sepa¬ 
rate  codes.  Creating  a  viable  environment  applica¬ 
tion  requires  the  integration,  testing,  and  tuning 
of  the  pieces — a  scientifically  and  technically  for¬ 
midable  task.  When  we  can  represent  each  piece 
as  a  component  with  a  standard  interface  and  be¬ 
havior,  then  integration — at  least  at  the  technical 
level — is  more  straightforward.  Similarly,  stan¬ 
dard  interfaces  help  foster  component  interoper¬ 
ability  and  component  use  in  different  contexts. 
This  is  a  primary  concern  for  modelers  because 
they’re  motivated  to  explore  and  maintain  al¬ 
ternative  algorithm  versions  (such  as  different 
implementations  of  the  atmosphere’s  governing 
fluid  equations),  whole  physical  domains  (such  as 
oceans),  parameteri nations  (such  as  convection 
schemes),  and  configurations  (such  as  stand-alone 
versions  of  physical  domains). 

ESMF  Components 

ESMF  has  two  types  of  components:  gridded 
components  (ESMF_GridComp)  represent  a  model’s 
scientific  and  computational  functions,  while  cou¬ 
pler  components  (ESMF_CplCoinp)  contain  the  op¬ 
erations  necessary  to  transform  and  transfer  data 
between  them.  Both  gridded  and  coupler  compo¬ 
nents  are  implemented  in  the  Fortran  interface  as 
derived  types  with  associated  modules.  Because 
ESMF*  doesn’t  currently  contain  prefabricated 
gridded  or  coupler  components,  users  must  write 
them.  The  ESMF  documentation  and  source  dis¬ 
tribution  provide  tools  and  examples  to  guide  us¬ 
ers  through  this  task. 

Each  major  physical  domain  in  an  F'arth  system 
model  is  represented  as  an  ESMF  gridded  com¬ 
ponent  with  a  standardized  calling  interface  and 
arguments.  Physical  processes  or  computational 
elements,  such  as  radiative  processes  or  I/O,  also 
can  be  represented  as  gridded  components.  ESMF' 
components  can  be  nested,  so  that  parent  compo¬ 
nents  can  contain  child  components  with  progres¬ 
sively  more  specialized  processes  or  refined  grids. 

As  a  model  steps  forward  in  time,  the  physi¬ 
cal  domains  represented  by  gridded  components 
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must  periodically  transfer  interfacial  fluxes.  The 
operations  necessary  to  couple  gridded  compo¬ 
nents  together  might  involve  data  redistribution, 
spectral  or  grid  transformations,  time  averaging, 
and  unit  conversions.  In  ESMF,  a  coupler  com¬ 
ponent  encapsulates  these  interactions.  Coupler 
and  gridded  components  share  the  same  standard 
interfaces  and  arguments.  The  interfaces’  key  data 
structure  is  the  ESMF_State  object,  which  holds 
the  data  to  be  transferred  between  components. 

Each  gridded  component  is  associated  with  an 
import  state  containing  the  data  required  for  it  to 
run,  and  an  export  state  containing  the  data  it  pro¬ 
duces.  Coupler  components  arrange  and  execute 
data  transfer  from  producer-gridded  components’ 
export  states  into  consumer-gridded  components’ 
import  states.  The  same  gridded  component  can 
be  a  producer  or  consumer  at  different  times  dur¬ 
ing  model  execution. 

There’s  no  single,  generic  coupler  component 
for  all  ESMF  applications.  Modelers  write  coupler 
component  internals  using  ESMh'  classes  bundled 
with  the  framework.  These  classes  include  meth¬ 
ods  for  time  advancement,  data  redistribution, 
interpolation  weight  calculation,  interpolation 
weight  application  through  a  sparse  matrix  multi¬ 
ply,  and  other  common  functions. 

Users  can  write  coupler  components  to  trans¬ 
form  data  between  a  pair  of  gridded  components 
or  use  a  single  coupler  component  to  couple  more 
than  two  gridded  components.  zMultiple  couplers 
can  be  included  in  a  single  modeling  application. 
'Fhis  is  a  natural  strategy  when  the  application  is 
structured  as  a  component  hierarchy.  Each  level 
in  the  hierarchy  usually  has  its  ow  n  set  of  coupler 
components. 

Design  Goals  and  Strategies 

Design  goals  for  ESMF  applications  include  the 

ability  to 

•  use  the  same  gridded  component  in  multiple 

contexts, 

•  swap  different  gridded  component  implementa¬ 
tions  into  an  application,  and 

•  assemble  and  extend  coupled  systems  easily. 

In  short,  the  goal  is  software  reuse  and  interoper¬ 
ability. 

One  design  pattern  that  addresses  these  goals 
is  the  mediator  pattern,  in  which  one  object  en¬ 
capsulates  how  a  set  of  other  objects  interact.^ 
The  mediator  serves  as  an  intermediary  and 
prevents  objects  from  referring  to  each  other  ex¬ 
plicitly.  ESMF  coupler  components  follow  this 


pattern.  It’s  an  important  aspect  of  the  ESMF 
technical  strategy  because  it  lets  users  deploy 
an  application’s  gridded  components  in  multi¬ 
ple  contexts — that  is,  it  lets  them  be  used  in 
different  coupled  configurations  without  chang¬ 
ing  the  source  code.  For  example,  a  user  might 
couple  the  same  atmosphere  to  an  ocean  in  a 
hurricane  prediction  model  and  to  a  data  assimi¬ 
lation  system  in  a  numerical  weather  prediction 
model. 

Another  mediator  pattern  advantage  is  that 
it  promotes  a  simplified  view  of  intcrcomponent 
interactions.  'Fhe  mediator  encapsulates  all  the 
complexities  of  data  transformation  between  com¬ 
ponents.  However,  this  can  lead  to  excessive  com¬ 
plexity  within  the  mediator  itself.^  One  approach 
to  addressing  this  is  to  create  multiple,  simpler 
coupler  components  and  predictably  embed  them 
in  a  hierarchical  architecture.  The  degree  to 
which  we  can  apply  a  hierarchical  approach  de¬ 
pends  on  the  intcrcomponent  interactions’  nature. 

Computational  environment  and  throughput 
requirements  motivate  a  different  set  of  design 
strategies.  ESMF  component  wrappers  must  not 
impose  significant  overhead  and  must  operate  effi¬ 
ciently  on  a  wide  range  of  computer  architectures, 
including  desktop  and  pctascalc  supercomputers. 
To  satisfy  these  requirements,  the  ESMF  soft¬ 
ware  relies  on  memory-efficient  and  highly  scal¬ 
able  algorithms,  such  as  that  by  Karen  Devine  and 
her  colleagues.*^  FISMF  runs  efficiently  on  tens  of 
thousands  of  processors. 

How  users  map  a  modeling  application’s  com¬ 
ponents  to  computing  resources  can  significantly 
impact  performance.  Strategies  vary  for  differ¬ 
ent  computer  architectures,  and  F.SMh'  is  flexible 
enough  to  support  multiple  approaches.  ESMF 
components  can  run  sequentially  (one  following 
the  other,  on  the  same  computing  resources),  con¬ 
currently  (at  the  same  time,  on  different  comput¬ 
ing  resources),  or  by  combining  these  execution 
modes.  Most  ESMF  applications  run  as  a  single 
executable — that  is,  all  components  are  combined 
into  one  program.  Starting  at  a  top-level  driver, 
each  level  of  an  ESMF  application  controls  the 
partitioning  of  its  resources  and  the  next  lower 
level’s  component  sequencing. 

As  we  now  describe,  these  goals  and  strategics 
have  been  implemented  in  various  Earth  system 
subdomains,  including  space  and  marine  weather 
and  the  coastal  watershed. 

Space  Weather  Modeling 

Giving  military  commanders  actionable  weather 
information  is  at  the  heart  of  the  Air  Force 
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Weather  Agency’s  mission.  Space  weather  is  no 
different.  To  give  commanders  ample  time  to 
prioritize  missions,  a  major  goal  of  space  weather 
is  to  forecast  for  the  120-hour  air-tasking-order 
cycle.  To  accomplish  this  in  the  ionosphere,  up¬ 
stream  information  is  required  about  both  solar 
irradiance  and  the  solar  wind,  which  drives  high- 
latitude  currents  that  affect  the  plasma  density  at 
high,  middle,  and  low  latitudes.  The  end  result 
is  longer,  more  accurate  forecasts  of  ionospheric 
conditions. 

The  Space  Weather  Modeling  System  (SWMS) 
is  a  BEI  project  that  couples  two  space  environ¬ 
ment  models  under  ESMF:  the  Hakamada- 
Akasofu-Fry  version  2  (HAFv2)  solar  wind  model 
and  the  global  assimilation  of  ionospheric  mea¬ 
surements  (GAlMl)  forecast  component.  The 
HAFv2  model  ingests  solar  observations  and  pro¬ 
vides  the  outputs  to  GAIMl  to  forecast  the  time- 
dependent  energy  input  into  the  high-latitudc 
ionosphere.  The  resulting  output  must  be  consis¬ 
tent  with  the  predcvelopmcnt  code,  scalable,  and 
portable.  The  coupled  HAFv2-GAIMl  will  show 
payoff  to  AFWA  operations  by  providing  the  first 
quantitative  forecasts  of  ionospheric  conditions 
that  extend  days  into  the  future. 

Hakamada-Akasofu-Fry  Version  2 
Exploration  Physics  International  (EX PI)  devel¬ 
oped  HAFv2  to  predict  solar  wind  conditions  at 
the  Earth  and  elsewhere  in  the  solar  system  days 
in  advance.^'^  The  HAFv2  model  is  designed  to 
track  interplanetary  disturbances’  progress  after 
solar  events.  Predicted  solar  wind  parameters  of 
speed,  density,  dynamic  pressure,  and  interplan¬ 
etary  magnetic  field  are  key  inputs  to  numerical 
prediction  models  that  forecast  near-Earth  space 
weather  disturbances  such  as  geomagnetic  storms, 
enhanced  energetic  particle  fluxes,  and  iono¬ 
spheric  disturbances. 

The  HAFv2  model  is  driven  by  solar  event 
reports  and  by  synoptic  solar  observations  in 
the  form  of  source  surface  maps  of  radial  mag¬ 
netic  fields  and  speeds  at  2.5  solar  radii. The 
source  surface  maps  arc  introduced  at  the  inner 
boundary  to  initialize  the  HAFv2  model  run. 
Event  reports  arc  converted  into  time-dependent 
perturbations  that  modulate  the  model’s  inner 
boundary.  Given  these  observation-based  inputs, 
HAFv2  predicts  a  time  series  of  solar  wind  val¬ 
ues  at  Earth.  Spacecraft  such  as  the  Advanced 
Composition  Explorer  located  in  a  stable  orbit 
upstream  (sunward)  of  the  Earth  provide  the 
ground-truth  measurements  for  comparison  with 
the  solar  wind  predictions. 


Global  Assimilation  of 
Ionospheric  Measurements 

The  GAlMv2.3  effort*^’**  merges  ionospheric  ob¬ 
servations  with  state-of-the-art  ionospheric  model 
results  using  a  Gauss-Markov  Kalman  Filter  to 
produce  a  real-time  weather  description  of  the 
ionosphere.  The  physics-based  ionosphere  model 
in  GAlMv2.3 — that  is,  its  GAIM 1  component — is 
the  Space  Environment  Corporation’s  ionosphere 
forecast  model.  The  IFM  is  a  global  ionosphere 
model  from  90  to  1,400  kilometers.  It  solves  for 
electron  and  ion  densities  as  well  as  electron  and 
ion  temperatures.^^  The  solar  wind  and  solar 
spectrum  are  the  primary  inputs  into  I  I'M. 

Coupling  the  Models 

GAIMv2.3  provides  a  24-hour  forecast  of  iono¬ 
spheric  conditions  assuming  persistence  of  the 
current  day’s  geophysical  conditions.  Without 
solar  inputs,  ionospheric  forecasts  relax  to  iono¬ 
spheric  climatology.  One-way  coupling  of  H  Alw2 
to  GAIMl  links  the  solar  storm  drivers  to  the  ion¬ 
ospheric  response.  H  AFv2  provides  the  solar  wind 
speed,  density,  and  magnetic  field  to  GAIMl, 
enabling  multiday  forecasts  of  ionospheric  elec¬ 
tron  density,  currents,  and  upper-atmosphere 
dynamics. 

Figure  I  shows  the  HAFv2  forecasts  from 
1-29  January  2000,  a  highly  active  space  weather 
period  at  the  11-ycar  solar  cycle’s  peak.  HAFv2 
simulates  the  timc-dcpcndcnt  interplanetary 
magnetic  field  (Figure  la)  that  results  from  the 
input  solar  conditions.  The  predicted  solar  wind 
quantities  at  h'arth  (Figure  Ib)  arc  fed  as  inputs 
to  GAIMl.  Figure  2  shows  the  GAIMI’s  sub¬ 
sequent  prediction  of  the  high-latitudc  response 
of  joule  heating  before  and  after  the  27-28 
January  storm’s  onset. 

SWMS  development  is  a  structured  project 
with  well-defined  milestones,  moving  from  par¬ 
tial  to  full  adoption  of  ESMF.  Vhc  SVV^MS’s 
coupled  HAFv2-GAIMI  components  incorpo¬ 
rate  the  necessary  infrastructure  and  superstruc¬ 
ture  to  achieve  full  ESMF  adoption.  I'hc  SWMS 
oversees  initialization,  running,  coupling,  and  fi¬ 
nalization  of  the  I  IAFv2  and  GAIMl  components 
via  ESMF  component  calls.  In  addition  to  cou¬ 
pling,  the  project  also  imposes  requirements  for 
portability,  scalability,  and  accuracy.  We  achieved 
HAFv2  and  GAIMl  model  scalability  through 
extensive  restructuring  and  implementing  paral¬ 
lel  algorithms. 

Bringing  the  HAF  and  GAIM  models  together 
in  SWMS  has  enabled  significant  improvements 
in  space  weather  data  processing  and  throughput. 


September/October  2010 


39 


160 

1 


26  Ian.  2000 
Time  =  18  UT 


Toward 
Away  1 3 

(»> 


2.0  AU 


Ecliptic  plane  IMF 


Solar  wind  parameters  at  LI  (GSM) 
Speed,  density,  and  IMF 


Month/Day  2000 


Figure  1.  Hakamada-Akasofu-Fry  version  2  (HAFv2)  solar  vvind  forecasts  from  1-29  january  2000.  (a)  HAFv2 
simulation  of  the  Interplanetary  magnetic  field  (IMF)  in  the  ecliptic  plane  to  2  astronomical  units  (AU)  for 
26  january  2000.  The  Earth's  orbit  is  at  1  AU;  its  location  is  shown  by  the  black  dot.  The  red  lines  represent 
outward-directed  regions  of  IMF,  while  the  blue  represent  inward-directed  regions,  (b)  HAFv2  simulation 
of  solar  wind  speed,  density,  and  IMF  magnitude  (fl),  north-south  angle  {0),  and  azimuthal  angle  (v?)  during 
1-29  january  2000. 
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Figure  2.  The  global  assimilation  of  Ionospheric 
measurements'  (CAIMl's)  HAFv2-driven  predictions 
of  the  high-latitude  response  of  joule  heating 
before  and  after  the  27-28  january  2000  storm's 
onset.  The  prediction  was  calculated  in  the 
northern  polar  region  using  the  HAFv2-CAIMl 
coupled  models.  The  joule  heating  magnitude  is  a 
key  component  of  the  ionospheric  forecast. 


Modifying  the  codes  for  the  high-performance 
computing  (HPC)  environment  brings  other  new 
capabilities,  including  the  ability  to 

•  ingest  diverse  data  sets  at  a  higher  resolution 
and  cadence, 

•  use  denser  computational  grids,  and 

•  perform  ensemble  forecasts. 

The  result  is  not  just  more  accurate  ionospheric 
forecasts  for  DoD  missions,  but  also  improved 


solar  wind,  geomagnetic,  and  thermospheric 
forecasts  for  the  DoD  and  other  government  and 
commercial  users. 

Weather  and  Marine  Prediction 

Given  the  continuing  global  warming  trend, 
there’s  a  pressing  need  to  better  understand  the 
interactions  between  the  atmosphere  and  ocean. 
How  these  two  systems  respond  together  to  the 
rising  temperature  dictates  our  ability  to  project 
future  climate  change  and  its  impact  on  shorter 
time-scale  prediction.  Coupling  different,  sophis¬ 
ticated  atmosphere,  ocean,  and  wave  models  to 
form  one  superior  system  is  therefore  an  important 
approach  for  capturing  many  physical  and  dynam¬ 
ical  processes  that  govern  the  air-sea  interaction. 

The  Naval  Research  Lal>oratory  (NRL)  Cou¬ 
pled  Ocean  and  Atmosphere  Mesoscalc  Predic¬ 
tion  System  (COAMPS)  is  a  high-resolution, 
fully  coupled  air- ocean -wave  system. As  Figure  3 
shows,  COAMPS’  ocean  circulation  model  is  the 
limited-area  version  of  the  Navy  (Coastal  Ocean 
Model  (NCOM)**^’* *^  and  it  can  incorporate  the 
Simulating  Weaves  Nearshore  (SWAN)  and 
Wavcwatch  III  wave  models.  Efforts  arc  under¬ 
way  to  also  include  the  Hybrid  Coordinate  Ocean 
Model  (HYCOM;  www.hycom.org). 

At  the  heart  of  COAMPS  is  a  driver/coupler 
that  controls  the  time-stepping  and  coordinates 
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field  exchange  between  components.  For  each 
component  pair,  the  coupler  computes  a  sparse 
matrix  that  combines  the  weights  for  interpolat¬ 
ing  between  grids  and  the  extrapolation  weights 
for  treating  land-sea  boundary  mismatches.  The 
ESMF  parallel  sparse  matrix  multiply  efficiently 
handles  the  grid  transformations  at  each  cou¬ 
pling  interval.  In  contrast  to  loosely  coupled 
systems,  the  coupling’s  scalability  and  efficiency 
allows  for  tight  model  integration.  The  coupling 
interval  can  be  as  small  as  the  least  common 
multiple  of  the  coupled  components’  time  steps. 
In  typical  applications,  with  coupling  intervals  of 
about  two  to  three  atmospheric  time  steps,  the 
coupling  overhead  is  less  than  1  percent  of  the 
overall  computation  time.  Background  ocean 
and  wave  components  are  included  to  support 
flexible  model  setup  and  improved  relocation 
capability. 

Development  of  the  fully  coupled  air-ocean 
system  was  completed  in  mid-2008  and  represents 
the  first  limited-area  weather  and  ocean  predic¬ 
tion  system  that  uses  ESMF  to  couple  air  and 
ocean  models.  Sue  Chen  and  her  colleagues  offer 
a  detailed  description  of  the  system  along  with 
results  of  two  test  cases.*^  The  hurricane  Katrina 
test  case  is  of  particular  interest  because  hurricane 
and  tropical  cyclones  have  a  tremendous  impact 
on  the  safety  of  coastal  communities  and  DoD 
operations. 

One  highlight  of  the  researchers’  study  was 
the  coupled  system’s  ability  to  simulate  realis¬ 
tic  hurricane  intensity  reduction  and  structural 
change  because  of  the  hurricane-induced  trail¬ 
ing  ocean  cold  wake.  Fhe  atmospheric  response 
to  the  ocean  cold  wake  was  significantly  reduced 
heat  and  moisture  fluxes  from  the  ocean  and  an 
increase  in  the  storm’s  flow  asymmetry.  Analysis 
of  the  ocean  temperature  budget  suggested  that 
within  the  ocean’s  mixed  layer,  vertical  advection 
(upwelling)  and  wind  mixing  contribute  equally  to 
the  cold  wake’s  generation  near  the  storm’s  center. 
Additional  cooling  found  to  arise  in  the  cold  wake 
along  its  right-front  quadrant  was  because  of  hori¬ 
zontal  advection  of  colder  water  forced  upward 
along  the  storm  track.  These  results  clearly  indi¬ 
cate  the  inadequacy  of  applying  ID  mixed-layer 
models — which  ignore  horizontal  advection — to 
capture  the  full  impact  of  the  air-ocean  interac¬ 
tion,  which  impacts  the  sea-surface-temperature 
cooling’s  development  and  structure  in  a  mature 
hurricane’s  wake. 

For  strong  wind  conditions,  the  air-sea  en¬ 
ergy  exchange  from  ocean  surface  waves  is  non- 
negligible.  Therefore,  accurately  depicting 


Figure  3,  The  Coupled  Ocean  and  Atmosphere 
Mesoscale  Prediction  System  (COAMPS)  integrates 
air,  ocean,  and  wave  models  for  weather  and 
marine  predictions.  The  system  includes  a  limited 
area  version  of  the  Navy  Coastal  Ocean  Model 
(NCOM)  and  can  incorporate  the  Simulating 
Waves  Nearshore  (SWAN)  and  Wavewatch  III  wave 
models. 

air-occan  interaction  in  a  coupled  system  must 
include  a  wave  model.  We’ve  extended  the  air- 
ocean  coupled  system  to  include  the  SWAN  wave 
model  to  study  the  air-sea  interaction  under  hur¬ 
ricane  conditions. 

Wc  ran  a  scries  of  sensitivity  tests  on  the  Ka¬ 
trina  test  case  to  investigate  the  impact  of  the 
atmospheric  wind,  sea-surface  height,  and  ocean 
current  influences  on  wave  growth.  Comparing 
the  significant  wave  height  with  several  buoys 
from  the  National  Buoy  Data  Center  suggests 
reasonable  agreement  with  the  observed  wave 
state.  As  Figure  4a  shows,  the  coupled  model  has  a 
longer  and  higher  wave  developed  along  the  storm 
front  quadrant.  Compared  to  the  run  without  the 
ocean  current  effect  on  waves,  using  the  ocean 
current  model  reduced  the  hurricane-induced  wave 
growth  in  the  storm  front  quadrant  (hTgure  4h). 
Although  the  precise  wave  feedback  to  the  atmo¬ 
spheric  and  ocean  models  is  still  in  basic  research 
and  has  yet  to  be  implemented,  our  results  do 
show  some  wave -growth  sensitivity  to  the  air- 
ocean-wave  coupling. 

The  NRL  in-house  research  and  development 
projects  have  already  experienced  a  significant 
impact  since  they  successfully  transitioned  to  the 
h’SMF-based  coupled  air-ocean  system.  NRL  sci¬ 
entists  have  applied  the  coupled  system  to  study 
and  validate  many  different  types  of  weather  and 
ocean  scenarios  that  are  strongly  influenced  by 
air-sea  interaction,  including 

•  the  cold  ocean  upwelling  along  the  US  west 

coast  and  South  Chile; 
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Figure  4.  The  impact  of  wind,  sea-surface  height,  and  ocean  currents  on  wave  growth,  (a)  There  were 
significant  wave  heights  (m)  and  10-meter  atmospheric  winds  (red  vector)  before  the  fully  coupled  model- 
simulated  hurricane  Katrina  made  landfall,  (b)  The  significant  difference  in  wave  height  between  model 
runs  with  and  without  current  input  to  the  wave  model. 


•  a  study  of  the  Kuroshio  extension  current  in  the 
Western  Pacific; 

•  an  analysis  of  the  Mistral  wind  jets  in  the 
Adriatic; 

•  Madden-Julian  Oscillation  studies  in  the 
Indian  Ocean;  and 

•  tropical  cyclones  in  the  Pacific,  Indian  Ocean, 
and  Atlantic  basins. 

NRL  scientists  can  also  seek  new  funding  op¬ 
portunities  not  previously  possible  before  the 
fully  coupled  mode  was  developed.  BFA's  air- 
occan-wavc  coupling  framework  is  a  foundation 
for  further  development  and  research  of  next- 
generation  limited-area  high-resolution  weather 
and  marine  prediction  models  for  the  US  Navy. 
The  project’s  new  technology  will  also  be  trans¬ 
ferred  to  non-DoD  partners.  The  COAMPS  cou¬ 
pler,  for  example,  will  be  used  as  a  prototype  in 
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Figure  5.  Coupled  watershed  and  coastal  ocean  model  grids  for  the 
Biscayne  Bay  region.  Coupling  the  watershed  model  (pWASH123D) 
with  the  coastal  ocean  model  (AdCirc)  improves  modeling  and 
prediction  of  hydrodynamics  in  these  areas. 


a  joint  university,  NOAA,  and  NRL  initiative  to 
build  a  unified  air-sea  interface  module  for  future- 
generation  hurricane  research  and  operational 
models  in  the  US. 

Coupled  Watershed 
Nearshore  Modeling 

Realistic  modeling  of  flow  and  transport  in  hy- 
drosystems  is  important  to  infrastructure  plan¬ 
ning,  environmental  remediation,  and  ecosystem 
restoration.  In  the  coastal  areas  and  in  estuaries, 
where  fresh  and  salt  waters  meet  and  interact,  the 
ability  to  simulate  coupled  salinity  transport  and 
density-dependent  flow  is  essential  for  accurately 
modeling  water  quantity  and  quality.  Although 
there  are  many  channel,  overland,  groundwater, 
watershed,  nearshore,  and  ocean  models  that  can 
compute  both  water  flow  and  salinity  transport, 
most  of  them  are  stand-alone  models  and  few  were 
designed  or  later  parallelized  for  11PC>.  As  a  re¬ 
sult,  their  applicability  to  real-world  problems  is 
restricted. 

To  improve  modeling  of  the  watershed  and 
coastal  regimes,  the  Army  I'mginecr  Research 
and  Development  Center  in  VTcksburg,  Mis¬ 
sissippi,  collaborated  with  the  NRL  at  the 
Stennis  Space  Center  to  couple  a  watershed 
model  (pVVASH123D)  with  a  coastal  oc'can  model 
(AdCirc).  Figure  5  shows  an  example  application 
of  the  coupled  watershed  nearshore  model  for 
Florida’s  Biscayne  Bay  region. 

I'he  pWASH123D^^’‘^  model  is  a  physically 
based  finite-element  numerical  model  that 
computes  watershed  systems’  water  flow  and  sim¬ 
ulates  them  as  combinations  of  II)  channel  net¬ 
works,  2D  overland  regimes,  and  3D  subsurface 
media.  The  interactions  bctw'cen  different  media 
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(between  \~  and  2D,  2-  and  3D,  and  1-  and  3D) 
impose  flux  continuity  and  state  variable  continu¬ 
ity  on  the  medium  interfaces.  The  pVVASH123D 
model  aims  to  efficiently  simulate  the  regional 
scale  of  real-world  problems  on  HPC  machines. 
VVe  implement  different  parallel  algorithms  and 
partitioning  strategies  in  different  components  to 
maintain  load  balance  and  reduce  communication 
overhead.  The  model  implemented  is  for  parallel, 
distributed  memory  platforms  and  consists  of  a 
mix  of  C/C++  and  Fortran  code. 

AdCirc  (www.adcirc.org)  is  a  coastal  circulation 
and  storm  surge  model  that  uses  finite-element 
computer  programs  for  solving  time-dependent, 
free  surface  circulation  and  transport  problems  in 
2D  and  3D.  The  model  implements  the  continu¬ 
ous  Galcrkin  finite-element  method  based  on  the 
generalized  wave-continuity  equation.  AdCirc  is 
written  in  Fortran  and  developed  for  parallel,  dis¬ 
tributed  memory  platforms;  typical  applications 
include 

•  modeling  tides  and  wind-driven  circulation, 

•  analyzing  hurricane  storm  surges  and  flooding, 

•  dredging  feasibility  and  material  disposal 

studies, 

•  larval  transport  studies,  and 

•  nearshore  marine  operations. 

One  challenge  this  project  faced  involved  data 
exchange  in  memory  between  two  models  us¬ 
ing  unstructured  meshes  written  in  different 
programming  languages.  When  this  project 
was  initiated,  ESMF  didn’t  have  functionality 
for  unstructured  meshes.  Fo  allow  the  water¬ 
shed  modeling  project  to  evolve  with  FSMF,  w^e 
decided  to  use  the  framework  to  construct  the 
components  and  manage  data  flow  to  and  from 
the  coupler.  In  the  coupler,  however,  we  used 
DBuilder**^  to  interpolate  between  the  two  model 
grids.  The  DBuildcr  toolkit  is  a  parallel  data 
management  library  for  scientific  applications 
that’s  part  of  the  pWASH123D  infrastructure. 
Developers  use  DBuilder  to  implement  paral¬ 
lel  versions  of  their  codes;  the  toolkit  provides  a 
simple  and  consistent  interface  that  hides  many 
of  the  programming  details  associated  with  do¬ 
main  partitioning,  parallel  data  management, 
domain  coupling,  and  invoking  parallel  linear 
.solvers.  Because  DBuilder  supports  coupling  in¬ 
dependent  domains  in  a  single  model,  adopting 
it  to  exchange  data  among  multiple  models  was  a 
straightforward  task. 

Coupling  uses  the  coastal  (shore)  line  as  the 
coupling  interface.  Figure  6  shows  a  side  view 


Figure  6.  A  side  view  of  the  interface  between  the 
pWASH123D  and  AdCirc  model  domains.  Here,  fl 
represents  the  boundary  flux  from  surface  flow,  and 
f2  represents  the  subsurface  flow. 

perpendicular  to  the  coupling  interface.  In  Figure  5, 
the  elevation  drop  on  the  AdCirc’s  side  of  the 
interface  boundary  represents  a  possible  scenario 
in  which  AdCirc  is  run  without  wetting  and  dry¬ 
ing  and,  hence,  imposes  a  minimum  water  depth. 
We  use  the  water  surface  elevation  simulated  by 
AdCirc  at  the  coupling  interface  as  a  Dirichlct- 
type  boundary  condition  for  computing  channel 
flow  and  overland  flow  in  the  surface  water  system. 
With  the  hydrostatic  assumption,  this  computed 
surface  elevation  is  distributed  to  the  subsurface 
nodes  beneath  the  coastal  line  for  computing  sub¬ 
surface  flow^ 

"The  boundary  flow  associated  with  all  the 
coupling  interface’s  pWASm23D  nodes  are 
computed  and  then  distributed  to  the  intcrfiice’s 
AdCirc  boundary  nodes  as  a  flux-type  boundary 
condition.  The  pWASm23D  boundary  nodal 
flow  includes  contributions  from  both  surface 
and  subsurface  flow.  In  Figure  6,  fl  represents 
the  boundary  flux  from  surface  flow,  and  f2  rep¬ 
resents  the  subsurface  flow.  In  reality,  part  of  f2 
passes  through  Zone  A  and  enters  and  exits  the 
AdCirc  model  across  the  dashed  line  as  sinks  and 
sources,  respectively.  In  this  case.  Zone  A  must 
be  included  in  the  modeled  domain.  T  he  current¬ 
coupling  approach,  however,  simplifies  this  situa¬ 
tion  by  setting  f2  the  subsurface  contribution  to 
only  the  coupling  interface’s  AdCire  boundary^ 
nodes. 

Wc  couple  the  models  in  a  concurrent  mode 
and  time  lag  the  boundary- forcing  fields.  At  each 
coupling  time  step,  we  use  the  computed  average 
boundary  flux  and  water  surface  elevation  from 
the  previous  coupling  time  step  at  the  coupling 
interface’s  boundary  conditions.  The  simulation 
proceeds  to  the  next  coupling  time  after  both 
models  have  fini.shcd  computing  the  current  cou¬ 
pling  time  step. 

The  pWASHI23D-AdCirc  model  is  the 
first  coupled  unstructured-mesh  model  us¬ 
ing  ESMF.  Developers  can  apply  the  model’s 
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features  and  eapabilities  to  eouple  struetured- 
and  unstruetured-mesh  models.  When  compared 
to  stand-alone  models,  the  coupled  model  exhib¬ 
its  more  accurate  boundary  conditions  applied  to 
the  interfaee  boundary  between  the  w'atershed 
and  the  coastal  models,  especially  when  surfacc- 
subsurface  and  watershed  coastal  ocean  interac¬ 
tions  are  signifieant  during  storm  events.  I'his 
better  models  the  nearshore  area’s  hydrosystem, 
improving  our  understanding  of  its  eomplex  in¬ 
teractions.  This,  in  turn,  benefits  the  construe- 
tion  and  verification  of  water  management  plans 
and  other  w'ater-related  environmental  issues. 
By  adding  salinity  and  reactive  transport  ea¬ 
pabilities  in  the  future,  the  eoupled  model  will 
serve  as  a  eutting-edge,  design-level  modeling 
tool  for  sustainable  hydrosystem  and  eeosystem 
restoration. 


In  addition  to  improving  ESMF,  weVe 
made  signifieant  steps  toward  the  goal  of 
integrating  environmental  modeling  ea¬ 
pabilities  into  ESMF  as  a  common  model¬ 
ing  infrastructure.  7'hrough  BEI’s  focus  projects, 
we’ve  demonstrated  payoff  for  DoD  environmen¬ 
tal  research  and  operations,  and  made  progress 
toward  the  overall  goal  of  a  w'hole- Earth  envi¬ 
ronment  modeling  capability.  However,  many 
challenges  remain.  To  achieve  interoperability  of 
Earth  system  components,  it’s  necessary  to  define 
a  common  physical  architecture — which  physical 
processes  each  component  contains  and  how  those 
components  are  interconnected — and  establish 
metadata  and  usage  conventions.  We’re  making 
progress  in  addressing  these  challenges  through 
participation  in  larger  community  efforts,  such  as 
the  Earth  System  Curator  project^^  and  The  Na¬ 
tional  Unified  Operational  Prediction  Capability 
project  (www.weather.gov/nuope). 
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